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ABSTRACT 

This paper examines the effect of a foraal evaluation 
process upon educational change — specifically, the adoption of 
innovative reading programs and practices--at different 
organizational levels of implementation. Because of the size ^f the 
project (62 school systems, 1t(2 designated schools, three sets of 
categorical funded projects) , the evaluation design is very simple. 
SPORES Drqanization as perceived by participants, self-reports of 
teacher reading instruction practices, and general professional 
development and teacher inservice activities vere measured through 
survey instruments. The evaluation describes SPOB in terms of the 
above data and correlates student reading gains vith these data plus 
other indicators of the degree to which the school has folloved 
SPOP*recommended standards and practices. The evaluation bad three 
positive uses in the SPOP program to date; bookkeeping, 
self-definition, and program change. Acceptance of the evaluation 
process and use of it or its measures appears to depend upon the 
extent to which the person or group has participated in developing 
and conducting the evaluation. Outcomes suggest there are benefits to 
be gained irrom involving clients in evaluation activities to the 
fullest extent. (Author/GK) 
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The purpose of this paper is to r^xamine the effect of a formal evaluation 
process upon educational change speci '-ically, the adoption of innovative 
reading programs and practices at different organizational levels of imple- 
mentation. Unlike the other projects discussed here today, Louisiana's Special 
Plan Upgrading Reading, or SPUR, Project involves a State Department of Education 
initiative carried out in a stepwise fashion through each level of the "organi- 
zation chart" of the state's educational system. The program is administered 
through the State Department to regional teams of professionals, and from them 
through the local school systems to ultimately reach individual schools and 
classroom teachers. This structure allows an examination of how change is brought 
about, and how the evaluation of change has an effect at each level of the 
educational system. This is the second year of SPUR's operation, and the first 
year the project is being evaluated. Therefore, this discussion is concerned 
with the effects of developing and establishing the evaluation process. 

Description of the Project . SPUR is a statewide, and completely state- 
funded, project to improve reading among K-8 students through the development 
of local resources. It follows the philosophy and methods of the federal 
Right-to-Read program, providing regional teams of reading specialists who 
work as change agents wich local school systems in developing central office 
staff and school principals as instructional leaders. These principals and 
central office staff cormiit themselves to developing a reading plan for each 
school system and to participating in inservice training which they are expected 
to carry back to the schools and classroom teachers. SPUR also funds three 
categories of competitive grant projects which feed into the educational delivery 
system. These are the university college of education projects, targeted toward 
improving preservice and graduate teacher education; education option projects 
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for students with special reading problems or needs; and parent/comnunity 
involvement projects. Within the core organizational framework of SPUR 
there are two major kinds of activities: the system-wide or regional inservice 
and direct services provided by SPUR Technical Assistants (the char^ge agents) 
to designated Demonstration or Practice Schools. 

Both the philosophy and the organization of SPUR have some inherent tensions 
that affect the project's implementation and its reaction to the evaluation 
process. The change agent philosophy has some contradictions. For one thing, 
the Technical Assistants are both service-oriented reading specialists and 
change agents. It is often difficult to refrain from providing a needed service 
and to convince systems to develop this service for themselves. For another, 
this change agentry role is unfamiliar in many cases to SPUR staff and Iccal 
education staff. Not knowing how to define SPUR*s role is a problem that is 
discussed later in this paper. 

The organization of SPUR is both bureaucratic and decentralized. There 
is an identifiable chain of comm'jni cation from the State Department through the 
field-based reg'^'onal SPUR Teams, which are comprised of a Team Leader and 
Technical Assistants who work directly with local systems and schools. However, 
each school system is independent and has a very limited contractual relationship 
with the state SPUR organization. The regional Teams come under stress because 
they are responsible to both the state SPUR staff and the local school systems. 
This organization affects the evaluation process in several ways. Participation 
in the evaluation process is completely voluntary at the local level, all data 
collection must be negotiated through each local school system, and SPUR staff 
are directly collecting virtually all of the needed information. 

spur's Reaction to the Evaluation Process . As stated earlier, this is 
the first year in which there has been any attempt at a formal evaluation. 
Summative results of this year's activities will not be available until the 
^ summer. Because of the size of the project — 62 school systems and 142 desig- 
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nated schools, plus three sets of categorical funded projects the evaluation 
design is very simple. The organization of SPUR as it is perceived by partici- 
pants, self-reports of teacher reading instruction practices, and general 
professional development and teacher inservice activities are measured through 
survey instruments. A subset of schools Is r»lso conducting classroom observations 
for student engagement and teacher practices, and pre- and post-testing students 
with the CTB/McGraw-Hill Prescriptive Reading Inventory. The evaluation will 
describe SPUR in terms of the data listed above and will correlate student 
reading gains with some of these data plus other indicators of the degree to which 
the school has followed SPUR-reconmended standards and practices. SPUR Directors 
and SPUR Teams are collecting all data. 

SPUR had no evaluation in its initial year of operation. The project also 
had not developed systematic recordkeeping procedures. The evaluation of SPUR is 
not completely voluntary. The state SPUR staff were directed by the legislature 
to collect evidence of the project's impact upon elementary reading achievement 
with the strong implication that future funding would be dependent upon this. 
The State Department of Education SPUR staff and the local school systems are 
relatively unfamiliar with evaluation, their experience being by and large limited 
to the monitoring required by federally funded projects. This has meant that 
project st^ff tend to discoun*: any information that cannot be expressed by 
numbers: they complain that, "It's all in my head, but I can't document it." 
Other experience has included the personnel evaluation system mandated by the 
Louisiana legislature in recent years, which has made it difficult for school 
system and school staff to recognize that the classroom observations will not be 
used to evaluate teachers or schools, and the state's equall; new student assess- 
ment testing which has led SPUR Teams to feel that their schools must do "well" 
on the evaluation's reading test. The last is the mo^t sen' jus problem for the 
evaluation. If the misunderstanding forces teachers to drill students on test 
objectives it could alienate school faculty and mask any possible relationship 
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between SPUR-endorsed programs and practices and reading achievement. 

State Department of Education SPUR staff entered the evaluation with some 
anxiety because they were unfamiliar with the evaluation process and because 
they do not expect SPUR to have any large inmediate effect upon student reading. 
Some goals, such as Improving university pre-service teacher education, are 
very long range indeed. 

The field-based SPUR Teams reviewed the evaluation design before the school 
year began, but did not fully realize the amount of work it would entail until 
they actually began administering tests and training for classroom observations. 
SPUR Directors, Superintendents of Schools, principals and teachers have recognized 
that cooperation is voluntary but have participated because they viewed this as 
necessary for the continuation of a program they valued or through personal 
commitments to SPUR staff. In some cases they have found the information such 
as objectives mastery reports from the pre-testing — useful. At this point 
local school staff generally view the evaluation process as an additional expendi- 
ture of teacher and student time with possible problems of confidentiality if 
school and teacher data should become public. Technical Assistants are faced 
with the dilemma of collecting complete information without jeopardizing their 
relationship with school staff. And, the concern for confidentiality will make 
it diffi^-ult to provide evaluation findings to system and school-level staff in 
a usable manner. Teachers can be given their own classroom data for comparison 
with the statewide report but the data cannot be reported at an aggregate higher 
than the classroom. Many systems have only one school participating in SPUR, 
therefore, sy*:cem or regional reports would almost surely identify individuals 
or schools. 

To sum up, the evaluation process is viewed in part as a threat. It has the 
potential to discredit SPUR despite the participants* "knowing" thac they are 
having a positive effect. The evaluation requires staff time which the staff 
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sees as better spent in pursuing project goaU, and the evaluation raises 
problems of confidentiality that could threaten the project's legitimacy. 
Some developing reactions are more positive. 

Uses of the Evaluation Process , The evaluation has had three positive 
uses in the SPUR program to date: bookkeeping, self-definition and program 
change. This last has been viewed favorably by the State Department of 
Education SPUR staff but raises obvious problems for the evaluation. 

The bookkeeping activities ranged from providing SPUR with very basic 
headcount information to providing assistance in the project's overall organi- 
zation of information. As a very simple example, the project had an extensive 
directory of schools participating in SPUR but had never tallied the number of 
schools, teachers or students this directory represented. The staff knew that 
there were variations from system to system in the manner in which services were 
delivered (practice schools versus different types of systemwide impact) but 
had not arranged these into a taxonomy. A second bookkeeping function was the 
awareness that there was some value in keeping records of project activities. 
State SPUR staff organized the previous year's files and are now able to locate 
what information was retained. Finally, having organized information has been 
of some use at the regional and school levels of SPUR. This is probably best 
exemplified in the use of the reading achievement test. After discussion with 
project staif , a test was selected which provides criterion-referenced information 
about student reading for teacher use and norm-referenced scores for evaluation 
purposes. The test publisher also provided a correlation of the test's objectives 
with the Louisiana Minimum Standards / Maximum Goals for Reading and correlations 
between the test objectives and coimonly used basal series. SPUR Technical 
Assistants and the teachers with whom they work used this information to come 
up with rough diagnoses of student performance (none of the matches were 
perfect) in terms of requirements for the Louisiana State Assessment test 
and the basal series which were their resources for meeting these requirements. 



This was not a new attempt, but it was a benefit to systems which had not had 
such correlations in the past and it allowed SPUR staff in different school 
systems to share efforts. This had been impossible in the past since systems 
use different standardized testing, testing schedules, and basals. 

A second use of evaluation processes was self-definition. This is not 
uncomncn in evaluations, but should be considered here a major benefit albeit 
an unplanned one. Considering the project's unfamiliarity with evaluation, 
this was the step that legitimized the SPUR evaluation with the staff. There 
was some clarification of goals in simply interviewing staff about the past 
year's activities and asking them how decisions had been made. Self-definition 
appeared more clearly when the State Department of Education staff participated 
in developing the survey instruments to measure teacher practices in reading 
and the organizational structure and functions of the project. The teacher 
practices instrument was developed by asking SPUR to develop a description of 
what the ideal reading teacher would do. This model was worked into a survey 
format with a great deal of discussion among the project staff. The discussion, 
in turn, forced the staff to organize the different individual oractices they 
had been promoting; in the past these had been treated as separate elements 
without the whole they comprised being expressed as a total system. The staff 
also saw similarities between different approaches being used in the eight 
regions and how these could be subsumed under a model for reading instruction. 
This outcome could be particularly useful if it is presented as an overall 
system for organizing the content of SPUR. 

The development of the survey measuring the assignment of responsibilities 
in SPUR and the perception of who actually carries them out provided self- 
definition from a management rather than a programmatic perspective. The 
functions in operating SPUR — management, inservice, development, etc. — were 
derived from the project's handbook. Review of responses to this instrument 
showed some confusion and disagreement on the part of the SPUR staff about who 



was actually to be assigned specific functions, which functions were to be 
delegated, and to whom. It also brought out, after discussion, the way in 
which the responsibilities for functions could shift frcm the beginning of 
a system's involvement in SPUR to its maturity as a system which had completely 
adopted the project. Again, there is no evidence to suggest how the final 
product of this survey will be used. It could be ignored, used to encourage 
recalcitrant systems to toe the line, or viewed as a model for different 
management structures. 

The processes of developing evaluation instruments and training project 
staff to administer them have also brought about some programmatic changes. 
This occurred somewhat in the standardized testing already noted. Placing 
students in the test brought about school- or grade-wide informal reading 
inventory usage in some cases and also reinforced correlation of the basal 
seriGs with the state's Mini'num Standards . These were more changes in emphasis 
than changes in kinds of activity. The latter appeared most clearly in the 
classroom observations for student engagement rates. 

Student engagement, or time-on-task, measures were initially requested 
as part of the evaluation by the SPUR staff. There was a general feeling that 
this would be a good predictor of student reading gain and that classroom 
management systems being promoted in several regions could be basically 
measured by looking at student engagement. This, more than any other activity, 
was developed with the participation of the field-based Team Leaders and required 
fairly intensive training and effort on their part and the part of the Technical 
Assistants. Th^ observation system chosen also required the participation of 
the teachers observed, who recorded pert of the observational information. 
Initially, there was a great deal of anxiety and resistance. Eventually, the 
Technical Assistants and many of the teachers liked it. The Technical Assistants 
reported that the observations allowed them to see patterns in teaching and 

classroom management of which they had Drevioij<:iv/ k^^ 

y nau^^reviousiy been unaware and forced them 
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to concentrate on student behaviors rather than observing the teacher alone 
Teachers In many cases appreciated being included in the diagnostic process 
and found the resulting information useful. Use of the classroom observation 
system plus the attention that student engagement has received recently in 
professional literature have added time-on-task to the content of SPUR. What 
was intended as an evaluation measure may be used as a program component next year. 

Summary . There are several conclusions to be drawn from the manner in which 
the SPUR project has used its evaluation. The first is that this use began 
before any data were collected or information provided. The final evaluation 
report will have to admit a very strong Heisenberg effect: the act of ir^easure- 
ment itself changed the phenomenon being observed. The second is that there 
is no evidence at this point of evaluation having a chilling effect upon the 
adoption of the SPUR prelect. This may occur if the evaluation findings have 
no effect upon the project's continuation or direction. It may also occur if 
too much energy is placed into "teaching to the evaluation" rather than pursuing 
the project's avowed goals. The very problems of gaining voluntary participation 
in the evaluation and using project participants ^s data collectors may nave 
prevented a chilling effect. The final conclusion is that acceptance 
evaluation process and use of the process or its measures appears to aep^^jocn 
the extent to which tht person or group has participated in developing ar 
conducting the evaluation. Information appears to be used by the people wrc 
collect it. Although the processes and products of evaluation may have very 
different effects, the outcomes discussed in this paper suggest that there are 
many benefits to be gained from involving clients in evaluation activities to the 
fullest extent possible. 
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